Perception based predictive tracking for head mounted displays

ABSTRACT

There is disclosed a method of and apparatus for predictive tracking for a head mounted display. The method comprises obtaining one or more three-dimensional angular velocity measurements from a sensor monitoring the head mounted display and setting a prediction interval based upon the one or more three-dimensional angular velocity measurements such that the prediction interval is substantially zero when the head mounted display is substantially stationary and the prediction interval increases up to a predetermined latency interval when the head mounted display is moving at an angular velocity of or above a predetermined threshold. The method further includes predicting a three-dimensional orientation for the head mounted display to create a predicted orientation at a time corresponding to the prediction interval, and generating a rendered image corresponding to the predicted orientation for presentation on the head mounted display.

NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. This patent document may showand/or describe matter which is or may become trade dress of the owner.The copyright and trade dress owner has no objection to the facsimilereproduction by anyone of the patent disclosure as it appears in thePatent and Trademark Office patent files or records, but otherwisereserves all copyright and trade dress rights whatsoever.

RELATED APPLICATION INFORMATION

This application is a continuation of U.S. patent application Ser. No.14/702,314, filed May 1, 2015, which is a continuation of U.S. patentapplication Ser. No. 14/285,470, filed May 22, 2014, Now U.S. Pat. No.9,063,330, which claims the benefit and priority of U.S. ProvisionalPatent Application Ser. No. 61/829,008, filed May 30, 2013, all of whichare hereby incorporated by reference.

BACKGROUND

Field

This disclosure relates to predictive motion tracking for a head mounteddisplay.

Description of the Related Art

Head mounted displays have long been used in virtual reality andaugmented reality systems. Virtual reality systems, typically, envelop awearer's eyes completely and substitute a “virtual” reality for reality.These virtual reality environments may be crude, either intentionally orthrough lack of capability of the virtual reality system. However,virtual reality environments may also be detailed, interactive and quitecomplex, involving virtual people, conversations and experiences. Themost obvious exemplar of a virtual environment may be a video gameinvolving a player character interacting with a game world. However,virtual environments need not be games and may, instead, be educationalexperiences, group activities (such as a tour of a historical site), ormerely sitting in a virtual room with an avatar representative of afriend and carrying on a conversation.

Augmented reality systems, in contrast, typically provide an overlaysemi-transparent or transparent screen or screens in front of a wearer'seyes such that reality is “augmented” with additional information,graphical representations, or supplemental data. Augmented reality may,for example, superimpose “virtual” people, items, cars, rooms, spaces,signs and other data over reality to a viewer. Simple augmented realitysystems may simply provide information regarding the scene or area beingviewed (e.g. temperature, upcoming appointments for a wearer, speed ofmovement, GPS location, etc.). More complex augmented reality systemsmay superimpose “virtual” tangible objects onto a scene, such as walls,artwork, individuals and similar elements. These may update, inreal-time so that images presented on the augmented reality displayappear to be present within a location to a wearer.

In either system, the movement of a wearer of such a headset may betracked in order to react to user movements and update the images beingpresented. This tracking utilizes sensors, such as gyroscopes,accelerometers, magnetometers, and, in some cases, cameras or colorsensors that generate data pertaining to position, motion, andorientation of a headset. This tracking data can be used to generateinformation such as angular velocity, linear acceleration, andgravitational data that may in turn be used to adjust the display of theheadset in response to wearer movement.

Predictive movement has been incorporated into virtual reality andaugmented reality headsets in the past. However, sample rates for thesensors identified above have typically been quite long—on the ordertens of milliseconds—relative to the acuity of human vision. Because thetime between samples is long, these predictions often result inso-called “overshoot” where a prediction overshoots the actual headposition and orientation or must be smoothed so severely to avoid otherproblems that they result in predictions sufficiently inaccurate thatmerely not predicting movement delivers better results.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is an overview of a virtual reality system.

FIG. 2 is a block diagram of a computing device.

FIG. 3 is a block diagram of a virtual reality headset.

FIG. 4 is a functional diagram of a virtual reality system.

FIG. 5 is a user wearing a virtual reality headset.

FIG. 6, made up of FIGS. 6A and 6B, is an example of jitter produced bypredictive motion tracking.

FIG. 7, made up of FIGS. 7A, 7B and 7C, is an example of predictivemotion tracking compared to actual motion.

FIG. 8 is a flowchart showing perceptually tuned filtering as applied toa predictive motion tracking process.

Throughout this description, elements appearing in figures are assignedthree-digit reference designators, where the most significant digit isthe figure number and the two least significant digits are specific tothe element. An element that is not described in conjunction with afigure may be presumed to have the same characteristics and function asa previously-described element having a reference designator with thesame least significant digits.

DETAILED DESCRIPTION

Dynamic application of motion prediction so as to take into accountvarious factors including jitter, latency and overall responsiveness ofthe virtual reality system can significantly reduce the problemsassociated with predictive movement tracking. For example, only applyingpredictive movement to headsets that are moving and only to a degreesuitable for the size of that movement provides better overallpredictive accuracy than merely enabling predictive tracking for allmovements and to the same degree (over the same prediction interval).

Similarly, smoothing, for example, regression-based averaging ofrecently detected movements may be applied much more aggressively whenmotion data indicates that a headset is substantially motionless, butmuch less aggressively when motion data indicates that a headset ismoving at a relatively high angular velocity or is increasing in angularacceleration. This is because human eyes are very sensitive to so-called“jitter” when the head is substantially motionless than when a humanhead is turning. Jitter occurs when a series of tiny slight variationsin sensor measurements causes the resulting rendered video to “bounce.”When a head is motionless, this jitter is highly-visible. When inmotion, jitter is almost completely unnoticed by human perception.

These types of dynamic scaling of both smoothing and motion predictionmay be referred to as perceptually tuned filtering because they relyupon discoveries related to human perception in conjunction with anunderstanding of the technical limitations of hardware available for usein virtual reality systems.

Description of Apparatus

Referring now to FIG. 1, an overview of a virtual reality system 100 isshown. The system includes a user system 110 and an environment server120 connected by a network 150, and a VR headset 130. A user 135 is alsoshown, though the user 135 may be considered separate from the system100.

The user system 110 is a computing device that connects to and interactswith the VR headset 130 in order to generate a virtual realityenvironment for display on the VR headset. The user system 110 may be,for example, a typical desktop computer, a video game console, such asthe Microsoft® Xbox®, the Nintendo® Wii® and Wii U® and Sony®PlayStation® consoles. The user system 110 may, in some cases, be amobile device, such as a mobile telephone, tablet computer, or otherhandheld or smaller computing device. The user system 110 may includetask-specific hardware, such as a video processor typically used bycomputing devices for rendering three-dimensional environments. Thevideo processor may be housed within a stand-alone video card or may beintegrated into a larger processor or system-on-a-chip.

Typically, the user system 110 will run software, including operatingsystem software and potentially virtual environment software thatgenerates images of a virtual environment for display on the VR headset130. The virtual environment software may be, for example, video gamesoftware that generates images of a three-dimensional world such that awearer of the VR headset 130 appears to be “in” the virtual environmentof the game world.

The environment server 120 is a computing device. The environment server120 may be optional depending upon the implementation of the virtualreality system 100. In instances in which the virtual environment 100 ismulti-user or multiparty, the environment server 120 may provide some ofthe data used by the user system 110 in order to render the virtualenvironment.

For example, the environment server 120 may maintain and communicate,via the network 150, data pertaining to other users simultaneouslyconnected to the virtual environment being rendered by the user system110. In this way, user avatars for a series of users of VR headsets,like VR headset 130, may appear within the virtual environment together.As one user “moves” within that environment, the environment server 120may receive that data and pass it on to user systems, like user system110, associated with the other users. Thus, the environment server 120may enable a group of users to experience a virtual environment togetherin real-time.

In some embodiments, particularly those that do not include anymulti-user or multiparty experiences, the environment server 120 may beunnecessary. In such instances, the user system 110 may take allresponsibility for rendering the virtual environment.

The VR headset 130 is a computing device, like the user system 110, butwith a few special characteristics. The VR headset 130 includes at leastone integrated display and includes one or more sensors that may be usedto generate motion, position, and/or orientation data for the VR headset130. As a wearer of the VR headset 130 moves, the sensors generatemotion, position, and/or orientation data and transmit that to the usersystem 110. The user system 110 returns one or more images to bedisplayed on the display integrated into the VR headset 130.

The VR headset 130 may be connected to the user system 110 by standardcomputer input and output interfaces such as USB®, HDMI, DVI, VGA, fiberoptics, DisplayPort®, Lightning® connectors, Ethernet or by a custominterfaces specifically designed for connecting the VR headset 130 to auser system 110. Typically, at least a part of these connections will berelatively high-throughput to ensure that rendered video data images aredisplayed with as little delay as possible on the integrated display. Insome cases, wireless connections, using 802.11 protocols, Bluetooth, orradio frequency may be used to transmit a portion of the data betweenthe user system 110 and the VR headset 130.

The VR headset 130 may integrate some or all of the functionality of theuser system 110. However, using current technology, these systems remaindistinct from one another primarily because the processing power andstorage necessary to render a convincing virtual environment, and toperform the motion extrapolation described herein, are better left tomore powerful and larger processors currently available only in largeror more expensive computing devices like the user system 110.

In some instances, such as when the user system 110 is a mobile devicesuch as a tablet or mobile phone, the sensors of the mobile device andthe display of the mobile device may take the place of all or a part ofthe VR headset 130 functionality. For example, the display of the mobiledevice may become the display of the VR headset 130.

The user 135 is a wearer of the VR headset 130. The user 135 wears theheadset in such a way that he or she can see the display integrated intothe VR headset 130. The user 135 may interact with other peripherals,such as a mouse, keyboard, headphones, speakers, microphones, hand-heldcontrollers or other, similar interactive components (not shown) tofurther direct the user system 110 as it renders a virtual environmentfor display on the VR headset 130.

The network 150 is a wired or wireless network that connects the usersystem 110 to the environment server 120. This network 150 may be orinclude the Internet or a local area network (LAN) or a wide areanetwork (WAN). The network 150 may utilize typical protocols such asTCP/IP, Ethernet, or other large-scale networking protocols in order tofacilitate communication between the user system 110 or 120. Customprotocols may be used. Communications from the environment server 120 tothe user system 110 (and other user systems) may use broadcast protocolsrather than protocols ensuring delivery. This may enable the environmentserver to continually update data on the user system 110, withoutwaiting for a plurality of user systems to confirm receipt of the lastdata.

Turning now to FIG. 2, a block diagram of a computing device 200 isshown which is representative of the server computers, client devices,mobile devices and other computing devices discussed herein. The usersystem 110 and environment server 120 are examples of computing devices.

The computing device 200 may include software and/or hardware forproviding functionality and features described herein. The computingdevice 200 may therefore include one or more of: memories, analogcircuits, digital circuits, software, firmware and processors. Thehardware and firmware components of the computing device 200 may includevarious specialized units, circuits, software and interfaces forproviding the functionality and features described herein.

The computing device 200 has a processor 210 coupled to a memory 212,storage 214, a network interface 216 and an I/O interface 218. Theprocessor may be or include one or more microprocessors, integratedspecific function processors (such as video processors), applicationspecific integrated circuits (ASICs), or systems on a chip (SOACs).

The memory 212 may be or include RAM, ROM, DRAM, SRAM and MRAM, and mayinclude firmware, such as static data or fixed instructions, BIOS,system functions, configuration data, and other routines used during theoperation of the computing device 200 and processor 210. The memory 212also provides a storage area for data and instructions associated withapplications and data handled by the processor 210.

The storage 214 provides non-volatile, bulk or long term storage of dataor instructions in the computing device 200. The storage 214 may takethe form of a disk, tape, CD, DVD, SSD (solid state drive) or otherreasonably high capacity addressable or serial storage medium. Multiplestorage devices may be provided or available to the computing device200. Some of these storage devices may be external to the computingdevice 200, such as network storage or cloud-based storage.

The network interface 216 includes an interface to a network such asnetwork 150 (FIG. 1).

The I/O interface 218 interfaces the processor 210 to peripherals (notshown) such as the VR headset 130, displays, keyboards, mice,controllers, USB devices and other peripherals.

FIG. 3 shows a block diagram of a virtual reality headset 300 having acomputing device substantially the same as that discussed above withrespect to FIG. 2. The elements discussed above with respect to FIG. 2will not be repeated here. This computing device of the virtual realityheadset 300 may be entirely or partially implemented as asystem-on-a-chip.

The processor 312 may be lower powered than that available to afull-size computing device. The memory 314, network interface 316,storage 318, and input/output interface 320 may be integrated into asingle package and may be designed for a small instruction set in orderto provide quick response times.

The sensors 322 are sensors used to generate motion, position, andorientation data. These sensors may be or include gyroscopes,accelerometers, magnetometers, video cameras, color sensors, or othermotion, position, and orientation sensors. The sensors 322 may alsoinclude sub-portions of sensors, such as a series of active or passivemarkers that may be viewed externally by a camera or color sensor inorder to generate motion, position, and orientation data. For example, avirtual reality headset may include on its exterior a series of markers,such as reflectors or lights (e.g., infrared or visible light) that,when viewed by an external camera or illuminated by a light (e.g.,infrared or visible light), may provide one or more points of referencefor interpretation by software in order to generate motion, position,and orientation data. In this sense, these markers are not “sensors,”but they make up a sub-part of a sensor system that is used to generatemotion, position, and orientation data.

The sensors 322 may operate at relatively high frequencies in order toprovide sensor data at a high rate. For example, sensor data may begenerated at a rate of 1000 Hz (or 1 sensor reading every 1millisecond). In this way, one thousand readings are taken per second.When sensors generate this much data at this rate (or at a greaterrate), the data set used for predicting motion is quite large, even overrelatively short time periods on the order of the tens of milliseconds.

The virtual reality headset includes a display buffer 324, where imagedata for display is stored immediately before it is presented on thedisplay, and may be or include the memory 314 available to the processor312. In this way, the display may function rapidly in response to inputfrom the processor 312.

The display (not shown) of the virtual reality headset 300 may be one ormore displays. The display may, for example, sit immediately in front ofthe eyes of a wearer of the VR headset 130. The display may, then, berelatively lower resolution while still filling a large visual area tothe wearer. The display displays the virtual environment to the weareras rendered by the user system 110.

Turning to FIG. 4, a functional diagram of a virtual reality system 400is shown. The virtual reality system 400 includes a user system 410, anenvironment server 420 and a VR headset 430. These may correspond to theuser system 110, environment server 120 and the VR headset 130 ofFIG. 1. A user 435, which is not a part of the system, is also shown.

The user system 410 includes an operating system 411, networkinput/output interface 412, environment software 413, virtual realitydrivers 414, and a motion predictor and smother 415.

The operating system 411 is the operating system of the user system 410.This may be, for example, a Windows®, OSX®, Linux or other operatingsystem if the user system 410 is a personal computer or similar device.The operating system 411 may be a proprietary system, such as theoperating systems employed in typical video game consoles. Stillfurther, the operating system 411 may be iOS®, Android® or a similarmobile device operating system if the user system 410 is a mobiledevice. The operating system 411 provides an environment for othersoftware, drivers and peripherals to function.

The network input/output interface 412 enables the user system 410 tocommunicate with the environment server 420 (when present). The networkinput/output interface 412 may include typical network drivers andsoftware particular to the environment software 413 that enables networkcommunication.

The environment software 413 is the software that operates inconjunction with the other components to create the environment that ispresented as virtual reality. For example, environment software may bevideo game software that generates a three-dimensional world forpresentation on the display of the VR headset 430. The environmentsoftware 413 may include specialized software to enable it to rendermultiple displays from different angles simultaneously-as is typical ina VR headset, two displays or two distinct images on a single displayare provided, one for each eye. The environment software 413 may beintegrated with software designed specifically to enable it to outputcontent suitable for the VR headset 430.

The environment software 413 need not be video game software, but may beany type of virtual reality or augmented reality environment software.This software may generate virtual reality locations including fictionallocations, historical locations, educational locations, or almost anyimmersive environment that simulates reality.

The virtual reality drivers 414 may be a software overlay between theoperating system 411 and environment software 413 and the VR headset430. The virtual reality driver 414 accepts input from the environmentsoftware 413 in particular and outputs data to the VR headset 430 in aform suitable for display by the VR headset 430. The virtual realitydrivers 414 may interact with the VR headset 430 on one or more levelsof abstraction below the software level in order to speed thetransmission and use of environment data to enable the VR headset 430 toreact quickly. This may be especially true when the user system 410 is amore powerful system (e.g. including a more powerful processor) than theVR headset 430. In this way, the VR headset 430 may accept more raw datathat may be directly displayed on the VR headset 430 without anysubstantial additional processing required by the VR headset 430.

The motion predictor and smoother 415 is software responsible forpredicting movement, orientation, and position of the VR headset 430.The motion predictor may be a part of the virtual reality drivers 414.

The motion predictor and smoother 415 may accept data from sensors inthe VR headset 430 in order to operate upon that data and to generate aprediction of VR headset 430 movement, orientation, and position. Themotion predictor and smother 415 may, for example, average angularvelocity measurements over a short time frame (e.g., ten measurementsover the last ten milliseconds) to derive a prediction about the futuremovement, orientation, and position. Alternatively, the motion predictorand smother 415 may differentiate over a series of angular velocitymeasurements to derive an angular acceleration and extrapolate into thefuture a predetermined time to predict a future movement, orientation,and position. The prediction process will be described more fully belowwith reference to FIG. 8.

In addition, the motion predictor and smoother 415 may perform“smoothing” operations on sensor data. In the rawest form, motion,orientation, and position data received from a series of sensors in theVR headset 430 may be slightly inaccurate. For example, with samplerates in the one per millisecond range, occasionally, tiny variations inthe orientation, movement, or position of the VR headset 430 may resultin incremental sensor readings that do not represent actual changes inmovement, orientation, or position.

For example, even if a user wearing the VR headset 430 is substantiallystill, the sensors may still report minor variations. Using this data,the images generated for display on the VR headset may include tinyvariations that are not representative of real changes in the VR headset430. As a result, so-called “jitter” in the display of the VR headset430 may be introduced. This appears to the user of the VR headset 430 asan image “bouncing” around in tiny increments in response to these tinyvariations in data received from the VR headset 430 sensors.

In response to this jitter and other similar errors, the motionpredictor and smoother 415 may also apply smoothing to the receiveddata. This smoothing may, for example, take a series of recent readingsfrom the sensors and average them to derive the data used to actuallygenerate the movement, orientation, and position data that is used tocreate the next image for display on the VR headset 430. Other smoothingmethods may also be used such as linear or nonlinear filtering wherebysensor data is extrapolated over a time period, while removinglikely-outlier data.

The environment server 420 includes an operating system 421, a networkinput/output interface 422, environment software 423, and an environmentdatabase 424. The operating system 421 and network input/outputinterface 422 serve substantially the same function as those describedfor the user system 410 above except on behalf of the environment server420. The operating system 421 is more likely to be an operating systemsupporting multiple user connections, for example via a network.Similarly, the network input/output interface 422 may also bettersupport multiple connections or be provided with faster connections to anetwork.

The environment software 423 serves a different purpose than theenvironment software 413 on the user system 410. The environmentsoftware is not actually responsible for rendering any three dimensionalworld for presentation to a user. Instead, it may maintain a state for aplurality of users connected to the environment server 420. For example,if hundreds of users are simultaneously connected to the environmentserver 420, the environment software 423 may maintain an environmentdatabase 424 that indicates the locations, actions, movements and otherdata for each of those users. The environment database 424 maydynamically-update and store those locations in order to transmit thatdata to user systems, like user system 410 so that the user system 410environment software 413 can update its visual display to incorporatethat data. This may be, for example, the location within the threedimensional environment rendered by the environment software of anavatar representative of another user.

The VR headset 430 may be the substantially the same as the VR headset130 in FIG. 1 and include the computing device 300 in FIG. 3. The VRheadset 430 may be worn by a user 435 in order to experience a virtualreality or augmented reality virtual environment.

Turning now to FIG. 5, a user 535 wearing a virtual reality headset 530is shown. The VR headset 530 is worn over the eyes of user 535. The user535 head may be considered at the center of a three-dimensional axiswith axes of pitch 540, roll 550 and yaw 560. Pitch 540 is an x axis,roll 550 is a z axis and yaw 560 is a y axis in a three-dimensionalCartesian coordinate system.

Movement of the VR headset 530 (user movement) may be expressed as pitch540, roll, 550 and yaw 560. One method for expressing athree-dimensional orientation uses quaternions. This is because anythree-dimensional orientation can be represented by a quaternionexpressed in the form q(v, θ)=(cos(θ/2), v_(x) sin(θ/2), v_(y) sin(θ/2),v_(z) sin(θ/2)) in which q(v, θ) represent the same rotation.Quaternions have the benefit of enabling manipulations of rotations withrelatively few parameters while preserving geometry (for example, theHaar measure) under algebraic operations, which is very useful forperforming prediction in three-dimensional space.

The Haar measure helps to ensure that multiple rotations of an object inthree-dimensional space remain related to one another. For example,applying a third rotation to two earlier rotations would, preferably,result in those two earlier rotations still being separated by the samedistance. Maintaining consistency with respect to the Haar measure helpsto ensure this and functions well in conjunction with quaternions.

FIG. 6, made up of FIGS. 6A and 6B, shows an example of jitter producedby predictive motion tracking. FIG. 6A shows a scene as two images 620and 622 presented on a VR headset. Two images 620 and 622 are shown, onefor each eye and from slightly different perspectives, in order tocreate the perception of depth in the scene. For example, the individual630 in the scene may be seen from slightly different angles in eachimage 620, 622. As a result, the mind perceives the individual 630having depth of field. This increases the immersion.

FIG. 6B shows a slightly different scene including images 640 and 642.Here, the individual 650 has “moved” upward in the scene. This type ofmovement may occur in situations in which predictive tracking is appliedand there are minor sensor errors and drift in the sensor detection. Forexample, a minor fluctuation in the sensors suggesting that a wearer'shead is “looking up”, may cause a predictive motion tracking system toover-extrapolate this motion data into a drastic movement that is notpresent.

In FIG. 6B, a minor fluctuation has been extrapolated to create motiondata for motion that is not present. The wearer's head is substantiallystationary. If this happens several times over the course of a fewseconds, the effect is drastic. Although the image is exaggerated sothat it is more easily recognizable in FIG. 6B, a wearer of a motionlessVR headset experiences even minor jitter as extremely non-immersivebecause a wearer's brain believes that it is stationary. Visuallyperceived movement data, even small jitter, that does not correspondwith actual head movement (or non-movement) does not match with thewearers remaining perception.

FIG. 7, made up of FIGS. 7A, 7B and 7C, shows an example of predictivemotion tracking compared to actual motion. FIG. 7A shows the same sceneas FIG. 6A with images 720, 722 and individual 730.

FIG. 7B is a scene rendered using predictive motion tracking toextrapolate a wearer's movement over a time frame into the future. Theimages 740, 742 are updated such that individual 750 appears to havemoved to the left. In visual perception, this is likely because thewearer began the process of turning his or her head to the right,causing the individual 750 to begin moving out of the field of view.

The predictive motion tracking has predicted the location of theindividual 750 (and the orientation of the entire scene) based uponmotion, orientation, and position data extrapolated from sensor datagenerated by sensors for the associated VR headset. For example, themotion, orientation, and position data may indicate that the VR headsetis moving along a particular path, and at a particular angular velocity(or potentially linear velocity or angular or linear acceleration). Thescene in 7B may be extrapolated (or predicted) over a given time frame.

FIG. 7C shows a scene including two images 760 and 762, includingindividual 770 that corresponds to the actual motion, orientation andposition of a VR headset. Careful examination of FIGS. 7B and 7Cindicates that the predicted motion was slightly incorrect.Specifically, the individual 750 did not move sufficiently far to theleft in the field of view as compared to the actual motion, orientation,and position shown in images 760 and 762. As a result, the individual770 is slightly further left in the scene. These errors are shown bycomparing predicted distances 744 and 746 with actual distances 764 and766.

Testing has indicated that when the processes described herein are used,these minor errors are virtually non-existent. However, predictivemotion tracking inherently includes some margin of error—it is aprediction of a wearer's movements before they occur. However, when theerrors are small, particularly in virtual reality systems, and becausethe time-frames these errors are visible are very fast, the wearer'smind typically does not carefully perceive the differences.

Description of Processes

Referring now to FIG. 8 a flowchart showing perceptually tuned filteringas applied to a predictive motion tracking process. The flow chart hasboth a start 805 and an end 895, but the process is cyclical in nature.In fact, the process is almost constantly iterating as a VR headset isactive on a wearer's head. For each frame of rendered video, the processmay iterate at least once and, potentially, several times.

First, sensor measurements are obtained from the sensors, includingsensors that may be in the VR headset, at 810. As discussed above, thesesensors may include accelerometers, gyroscopes, magnetometers, active orpassive markers and cameras. The cameras may be mounted on the headsetwith markers mounted in the surrounding environment or markers may bemounted on the headset and one or more external cameras may track themarkers. The function of these sensors is to provide accurate data fromwhich predictions of motion, orientation and position may be derived.Wherever and whatever the sensors, the sensors provide data to, forexample, the user system to begin the process of perception basedprediction.

Next, sensor fusion is performed at 820. This process may be complex ormay be simple. In a simple system, the raw data measurements (forexample, acceleration and velocity from an accelerometer) may merely becombined and standardized into a form suitable for use by environmentsoftware in rendering a three dimensional environment. In more complexsystems, the data may be “smoothed,” as discussed above, and dataextrapolating potential predictions for headset motion may be generatedat the sensor fusion stage and passed along to subsequent stages.

A determination is made, for example using the virtual reality drivers414, whether the headset is turning at 825. This determination isrelatively simple. A series of angular velocity measurements may begenerated over a short time period (e.g. 10 milliseconds) by thesensors. Those measurements may be averaged to determine if the angularvelocity exceeds a predetermined threshold, for example, 1-5 degree persecond. In this way, a relatively minor movement (or a small number ofsensor errors) will not be perceived as “movement” of the headset.

A human head wearing a VR headset cannot remain perfectly still, but amovement threshold can be set in such a way that the determinationwhether a headset is turning 825 can be made. A non-turning head issubstantially stationary. As used herein the phrase “substantiallystationary” means that the headset is not in the process of a head tumof a sufficient degree that the virtual reality system should re-renderthe associated scene to reflect that movement. Specifically,“substantially stationary” does not mean absolutely zero movement of theVR headset, but does mean “stationary” to the extent that a human,without attempting to move, can hold his or her head “still.”

If the headset is turning, then smoothing is disabled at 830. Asdiscussed somewhat above, smoothing is primarily directed tocounteracting sensor artifacts, minor movements, and sensor drift(linear or exponential drift in accuracy of sensors) so that these donot harm the experience of a wearer by producing jitter or randommovement of the rendered scene. After some study of human perception, ithas been discovered that individuals perceiving virtual realityenvironments do not experience jitter at all when their heads areturning. In particular, the mind either ignores jitter or is used toexperiencing some irregularity in a visual scene as a head turns andautomatically compensates.

As a result, if the headset is turning, smoothing may be disabled, atleast temporarily. This frees up some processor cycles to work on otherproblems, but more importantly, it ensures that any predictive motiontracking is not hindered by any pre- or post-calculation smoothing ofthe data. Heavy smoothing can cause rendered frames to appear to lagbehind actual movements, for example, by over-weighting prior dataindicating smaller (or no) movement or can result in overshoot. Thus,when the head is turning, no detrimental effect for a wearer isrecognized by disabling smoothing, and the benefit is more responsivescene rendering as the headset turns.

Next (or substantially simultaneously), an angular velocity of theheadset is determined at 840. This determination may be a singlemeasurement or single data set generated by the sensor fusion at 820.Preferably, the angular velocity is determined using one of a number ofoptions from an average of a predetermined prior set of angular velocitymeasurements (e.g. the most recent 10 measurements) to a weightedaverage of a similar set of angular velocity measurements, withadditional weighting applied to the most recent set of measurements(e.g. the most recent 1-3 measurements) to a determination of the linearacceleration by differentiation and extrapolation of that acceleration(or deceleration as the case may be) over an interval.

The next step is to set the prediction interval over which the motionprediction will take place at 850. As used herein, the “predictioninterval” is the time over which the prediction of motion is made. Forexample, if the prediction interval is 40 milliseconds, the predictivemotion tracking predicts the motion, orientation, and position of theheadset at the end of that prediction interval, 40 milliseconds fromnow. If the prediction interval is zero, the predictive motion trackingpredicts the motion, orientation, and position of the headset at thepresent time (effectively providing no predictive motion tracking).Including a prediction interval enables the predictive motion trackingsystem to dynamically respond to the size and scope of headsetmovements.

This prediction interval may be quite small, and is, for smallmovements. For example, movements on the order of 10 degrees/second mayhave a prediction interval set very small, for example to 5milliseconds. In such situations, the angular velocity measurement,determined by whatever method, are applied over that time period andprovided in place of actual data for that time period, to a user systemfor rendering of the associated video frames. The next frame of renderedvideo is delivered based upon the resulting prediction.

In situations in which the angular velocity measurement is quite large,relatively speaking (e.g., on the order of 500 degrees/second) theprediction interval over which the system predicts the movement is mademuch larger. For example, fast movement causes the system to set theprediction interval to be between 30 and 60 milliseconds. In thesesituations, the motion prediction data that is passed to a videorenderer predicts a wearer's motion, orientation, and position at theend of that prediction interval and delivers the next frame of renderedvideo based upon that prediction.

In well-studied systems, the latency interval may be well known. Thephrase “latency interval” is the time from when motion data is generatedby the sensors to the delivery of a rendered frame of videoincorporating that motion data. In the present system, that latencyinterval is on the order of 30 to 60 milliseconds and appears totypically be approximately 40 milliseconds.

This latency interval may be used as an upper bound for the predictioninterval. So, the prediction interval may not exceed the latencyinterval. This is because as a user moves his or her head while wearingthe headset, the next rendered frame of video incorporating that datawill appear at the next latency interval.

Accordingly, predicting beyond that time frame is unnecessary and morelikely to result in overshoot (rendering too great a movement or a stopwhen movement was just beginning) and disconnection from actual wearermovement. Similarly, additional data will be available, in the systemdescribed, tens of additional motion, orientation, and position datapoints that can be used to render the next frame of video. Predictingmotion up to the latency interval has the benefit of attempting topredict actual user movement with the next moment that user will see aframe of rendered video.

The process may work, for example, such that a determination is made asto the angular velocity of a given wearer movement. When the angularvelocity above a threshold rate, for example 0.3 radians per second,then the time over which prediction is increased linearly up to thelatency interval. This increase in the prediction interval may be, forexample, +2 milliseconds per radian per second. In this way, a headsetturning at 1 radians per second, would utilize a prediction interval of2 milliseconds in the future. A headset turning at 10 radians per secondwould utilize a prediction interval of 20 milliseconds in the future.Other linear increase slopes may be used, up to a maximum predictioninterval equal to the latency interval (e.g. 40 milliseconds into thefuture).

Thus, if the angular velocity is substantially zero (e.g. 0 to 0.03radians per second) the orientation in three dimensions is derived usingsmoothing and reliant upon the current position and orientation (withoutprediction). If the angular velocity is greater than substantially zero,then prediction algorithms are used and smoothing is disabled. Finally,the position (place in three-dimensional space) is derived, regardless,using the last measured position, plus the linear velocity over theselected possible prediction interval.

To summarize, the movement of the headset is measured to determinewhether or not movement (angular velocity) of the headset issubstantially zero (0 to 0.3 radians per second). If so, smoothing ofthe data is enabled and the prediction interval is set to substantiallyzero (or zero). If the headset is moving (angular velocity) at a ratelarger than substantially zero (greater than 0.3 radians per second),then smoothing is disabled and the prediction interval is set. If themovement is increasing in angular velocity, then the prediction intervalrises by a prediction interval variance (e.g., 2 milliseconds per radianper second) per reading up to a maximum of the latency interval (e.g.,40 milliseconds).

The increase in the prediction interval (up to the maximum of thelatency interval) may be linear, corresponding to linear movement of aheadset. As a result, faster movement is compensated for by linearincrease in prediction interval as described above. Alternatively, aso-called smoothstep algorithm may be used. Smoothstep algorithms takethe generalized form smoothstep(t)=3t²−2t³ where t is the variable to besmoothed and the results of the generalized form fall between 0 and 1.Smoothstep has the benefit of being sloped toward zero at both the highand low end. In this way, the prediction differs from linearinterpolation by being “rounded off” at the ends. This appears to awearer of such a headset relying upon a prediction interval in that theprediction more slowly approaches the maximum or minimum withoutimmediate or abrupt stops or starts at either end. As a result, theprediction interval can “smoothly” approach zero or approach the maximumlatency interval.

If the headset is not turning at 825, indicating that the user's head issubstantially still, then smoothing may be enabled at 860. As discussedabove, smoothing is most helpful when the headset is not moving or ismaking very small movements. In particular, it helps to avoidunnecessary or over-exaggerated movement of the scene, such as jitter,when a headset is substantially still. As such, enabling smoothing whena headset is substantially still will aid in reducing these issues.

Similarly, the prediction interval is set to substantially zero at 870.When the head is virtually non-moving, aggressive prediction over anyinterval is not necessary. Some predictive motion tracking may beapplied, but to the extent that it is, the interval over which it isapplied is set to a very short time frame, such as 5 milliseconds orless. In some cases, the prediction interval may be set to zero. Thisfurther helps to avoid predictive motion tracking that may exaggeratesmall movements, particularly if the prediction interval is set at ornear the latency interval.

Next, whether the headset is turning or not at 825, and after settingthe prediction interval at 850 and 870, a prediction of athree-dimensional orientation and angular velocity are made at 880. Ineffect, this predicts the motion, orientation and position of theheadset at the end of the prediction interval. This may be expressed, asdescribed above, as a quaternion, but may be expressed in any number ofmethods.

Finally, a display is updated at the next rendered video frame tocorrespond to the predicted motion, orientation, and position based uponthe prediction and the prediction interval at 890. As a result, if theprediction interval was 40 milliseconds, the next rendered frame ofvideo delivered to the VR headset will be rendered as though thewearer's head was at a predicted position 40 milliseconds from now. Ifthe prediction interval was zero milliseconds, then the next renderedframe of video delivered to the VR headset will be rendered as thoughthe wearer's head was at a predicted position at the present time (noprediction is necessary, actual data is available). So, no prediction ora very limited prediction is provided.

Both of these options and extremes reflect the fact that the head isturning or is not turning at 825. Once the display is updated, theprocess ends. However, as described above, the process repeats for everysensor update or for every interval of sensor updates (e.g. every 10 or20 sensor updates). In this way, the next frame of rendered video may beprovided, either with or without smoothing and with or without aprediction interval at or near a latency interval.

Closing Comments

Throughout this description, the embodiments and examples shown shouldbe considered as exemplars, rather than limitations on the apparatus andprocedures disclosed or claimed. Although many of the examples presentedherein involve specific combinations of method acts or system elements,it should be understood that those acts and those elements may becombined in other ways to accomplish the same objectives. With regard toflowcharts, additional and fewer steps may be taken, and the steps asshown may be combined or further refined to achieve the methodsdescribed herein. Acts, elements and features discussed only inconnection with one embodiment are not intended to be excluded from asimilar role in other embodiments.

As used herein, “plurality” means two or more. As used herein, a “set”of items may include one or more of such items. As used herein, whetherin the written description or the claims, the terms “comprising”,“including”, “carrying”, “having”, “containing”, “involving”, and thelike are to be understood to be open-ended, i.e., to mean including butnot limited to. Only the transitional phrases “consisting of” and“consisting essentially of”, respectively, are closed or semi-closedtransitional phrases with respect to claims. Use of ordinal terms suchas “first”, “second”, “third”, etc., in the claims to modify a claimelement does not by itself connote any priority, precedence, or order ofone claim element over another or the temporal order in which acts of amethod are performed, but are used merely as labels to distinguish oneclaim element having a certain name from another element having a samename (but for use of the ordinal term) to distinguish the claimelements. As used herein, “and/or” means that the listed items arealternatives, but the alternatives also include any combination of thelisted items.

What is claimed is:
 1. A method comprising: determining a firstthree-dimensional orientation of a head mounted display, the headmounted display comprising one or more inertial measurement units(“IMUs”) and one or more markers on an outside surface of the headmounted display; detecting a movement of the head mounted display;obtaining, from the one or more IMUs, first data representative of thedetected movement of the head mounted display; obtaining, from one ormore cameras external to the head mounted display, second datarepresentative of the detected movement of the head mounted displaybased on a movement determined by the one or more cameras of the one ormore markers; predicting a second three-dimensional orientation of thehead mounted display based on both the first data representative of thedetected movement and the second data representative of the detectedmovement; and generating a rendered image corresponding to the predictedsecond three-dimensional orientation of the head mounted display forpresentation by the head mounted display.
 2. The method of claim 1,wherein at least one of the one or more IMUs comprises one of: agyroscope, an accelerometer, and a magnetometer.
 3. The method of claim1, wherein at least one of the one or more markers comprises one of: areflector and a light.
 4. The method of claim 1, wherein at least one ofthe cameras external to the head mounted display is configured to detectone or both of infrared light and visible light from one or more of themarkers.
 5. The method of claim 1, wherein the one or more cameras areconfigured to provide a detected location of the one or more markers toa movement engine for use as points of reference, the movement engineconfigured to determine a movement of the one or more markers based onthe detected locations and to generate the second data representative ofthe detected movement of the head mounted display based on thedetermined movement of the one or more markers.
 6. The method of claim1, wherein predicting the second three-dimensional orientation of thehead mounted display comprises determining an angular velocity of thehead mounted display based on the first data representative of thedetected movement and the second data representative of the detectedmovement.
 7. The method of claim 6, wherein predicting the secondthree-dimensional orientation of the head mounted display furthercomprises determining a prediction interval of time based on thedetermined angular velocity of the head mounted display.
 8. A apparatuscomprising a non-transitory computer-readable storage medium storingexecutable computer instructions, and further comprising a hardwareprocessor, the hardware processor configured to execute the computerinstructions to cause the processor to: determine a firstthree-dimensional orientation of a head mounted display, the headmounted display comprising one or more inertial measurement units(“IMUs”) and one or more markers on an outside surface of the headmounted display; detect a movement of the head mounted display; obtain,from the one or more IMUs, first data representative of the detectedmovement of the head mounted display; obtain, from one or more camerasexternal to the head mounted display, second data representative of thedetected movement of the head mounted display based on a movementdetermined by the one or more cameras of the one or more markers;predict a second three-dimensional orientation of the head mounteddisplay based on both the first data representative of the detectedmovement and the second data representative of the detected movement;and generate a rendered image corresponding to the predicted secondthree-dimensional orientation of the head mounted display forpresentation by the head mounted display.
 9. The apparatus of claim 8,wherein at least one of the one or more IMUs comprises one of: agyroscope, an accelerometer, and a magnetometer.
 10. The apparatus ofclaim 8, wherein at least one of the one or more markers comprises oneof: a reflector and a light.
 11. The apparatus of claim 8, wherein atleast one of the cameras external to the head mounted display isconfigured to detect one or both of infrared light and visible lightfrom one or more of the markers.
 12. The apparatus of claim 8, whereinthe one or more cameras are configured to provide a detected location ofthe one or more markers to a movement engine for use as points ofreference, the movement engine configured to determine a movement of theone or more markers based on the detected locations and to generate thesecond data representative of the detected movement of the head mounteddisplay based on the determined movement of the one or more markers. 13.The apparatus of claim 8, wherein predicting the secondthree-dimensional orientation of the head mounted display comprisesdetermining an angular velocity of the head mounted display based on thefirst data representative of the detected movement and the second datarepresentative of the detected movement.
 14. The apparatus of claim 13,wherein predicting the second three-dimensional orientation of the headmounted display further comprises determining a prediction interval oftime based on the determined angular velocity of the head mounteddisplay.
 15. A non-transitory computer-readable storage medium storingexecutable computer instructions that, when executed by a processor, areconfigured to perform steps comprising: determining a firstthree-dimensional orientation of a head mounted display, the headmounted display comprising one or more inertial measurement units(“IMUs”) and one or more markers on an outside surface of the headmounted display; detecting a movement of the head mounted display;obtaining, from the one or more IMUs, first data representative of thedetected movement of the head mounted display; obtaining, from one ormore cameras external to the head mounted display, second datarepresentative of the detected movement of the head mounted displaybased on a movement determined by the one or more cameras of the one ormore markers; predicting a second three-dimensional orientation of thehead mounted display based on both the first data representative of thedetected movement and the second data representative of the detectedmovement; and generating a rendered image corresponding to the predictedsecond three-dimensional orientation of the head mounted display forpresentation by the head mounted display.
 16. The non-transitorycomputer-readable storage medium of claim 15, wherein at least one ofthe one or more IMUs comprises one of: a gyroscope, an accelerometer,and a magnetometer.
 17. The non-transitory computer-readable storagemedium of claim 15, wherein at least one of the one or more markerscomprises one of: a reflector and a light.
 18. The non-transitorycomputer-readable storage medium of claim 15, wherein at least one ofthe cameras external to the head mounted display is configured to detectone or both of infrared light and visible light from one or more of themarkers.
 19. The non-transitory computer-readable storage medium ofclaim 15, wherein the one or more cameras are configured to provide adetected location of the one or more markers to a movement engine foruse as points of reference, the movement engine configured to determinea movement of the one or more markers based on the detected locationsand to generate the second data representative of the detected movementof the head mounted display based on the determined movement of the oneor more markers.
 20. The non-transitory computer-readable storage mediumof claim 15, wherein predicting the second three-dimensional orientationof the head mounted display comprises determining an angular velocity ofthe head mounted display based on the first data representative of thedetected movement and the second data representative of the detectedmovement.
 21. The non-transitory computer-readable storage medium ofclaim 20, wherein predicting the second three-dimensional orientation ofthe head mounted display further comprises determining a predictioninterval of time based on the determined angular velocity of the headmounted display.